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ABSTRACT 

Applications ranging from algorithmic trading to scientific 
data analysis require realtime analytics based on views over 
databases that change at very high rates. Such views have 
to be kept fresh at low maintenance cost and latencies. At 
the same time, these views have to support classical SQL, 
rather than window semantics, to enable applications that 
combine current with aged or historical data. 

In this paper, we present viewlet transforms, a recursive 
finite differencing technique applied to queries. The viewlet 
transform materializes a query and a set of its higher-order 
deltas as views. These views support each other's incremen- 
tal maintenance, leading to a reduced overall view mainte- 
nance cost. The viewlet transform of a query admits efficient 
evaluation, the elimination of certain expensive query oper- 
ations, and aggressive parallelization. We develop viewlet 
transforms into a workable query execution technique, present 
a heuristic and cost-based optimization framework, and re- 
port on experiments with a prototype dynamic data man- 
agement system that combines viewlet transforms with an 
optimizing compilation technique. The system supports tens 
of thousands of complete view refreshes a second for a wide 
range of queries. 

1. INTRODUCTION 

Data analytics has been dominated by after-the-fact ex- 
ploration in classical data warehouses for multiple decades. 
This is now beginning to change: Today, businesses, engi- 
neers and scientists are increasingly placing data analytics 
engines earlier in their workflows to react to signals in fresh 
data. These dynamic datasets exhibit a wide range of up- 
date rates, volumes, anomalies and trends. Responsive an- 
alytics is an essential component of computing in finance, 
telecommunications, intelligence, and critical infrastructure 
management, and is gaining adoption in operations, logis- 
tics, scientific computing, and web and social media analysis. 

Developing suitable analytics engines remains challenging. 
The combination of frequent updates, long-running queries 
and a large stateful working set precludes the exclusive use 
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of OLAP, OLTP, or stream processors. Furthermore query 
requirements on updates often do not fall singularly into 
the functionality and semantics provided by the available 
technologies, from CEP engines to triggers, active databases, 
and database views. 

Our work on dynamic data management systems (DDMS) 
in the DBToaster project [3, 19, 18] studies the foundations, 
algorithms and architectures of data management tools de- 
signed for large datasets that evolve rapidly through high- 
rate update streams. A DDMS focuses on long-running 
queries as the norm, alongside sporadic exploratory query- 
ing. A guiding design principle for DDMS is to take full 
advantage of incremental processing techniques that max- 
imally reuse prior work. Incremental computation is cen- 
tral to both stream processing to minimize work as windows 
slide, and to database views with incremental view mainte- 
nance (IVM). DDMS aim to combine some of the advantages 
of DBMS (expressive queries over both recent and histori- 
cal data, without the restrictions of window semantics) and 
CEP engines (low latency and high view refresh rates). 

An example use case is algorithmic trading. Here, strategy 
designers want to use analytics expressible in a declarative 
language like SQL on order book data in their algorithms. 
Order books consist of the orders waiting to be executed at a 
stock exchange and change very frequently. However, some 
orders may stay in the order book relatively long before they 
are executed or revoked, precluding the use of stream engines 
with window semantics. Applications such as scientific sim- 
ulations and intelligence analysis also exhibit entities which 
capture our attention for widely ranging periods of time, 
resulting in large stateful and dynamic computation. 

The technical focus of this paper is on an extreme form 
of incremental view maintenance that we call higher-order 
IVM. We make use of discrete forward differences (delta 
queries) recursively, on multiple levels of derivation. That 
is, we use delta queries ("first-order deltas") to incremen- 
tally maintain the view of the input query, then materialize 
the delta queries as views too, maintain these views using 
delta queries to the delta queries ("second-order deltas"), 
and continue alternating between materializing views and 
deriving higher-order delta queries for maintenance. The 
technique's use of higher-order deltas is quite different from 
earlier work on trading off choices in which query subex- 
pressions to materialize and incrementally maintain for best 
performance [27]. Instead, our technique for constructing 
higher-order delta views is somewhat reminiscent of discrete 
wavelets and numerical differentiation methods, and we use 
a superficial analogy to the Haar wavelet transform as mo- 
tivation for calling the base technique a viewlet transform. 

Example 1 Consider a query Q that counts the number 
of tuples in the product of relations R and S. For now, 
we only want to maintain the view of Q under insertions. 



968 



Denote by A_r (resp. As) the change to a view as one tuple 
is inserted into R (resp., S). Suppose we simultaneously 
materialize the four views 

• Q (0-th order), 

• ArQ = count(S') and AsQ = count(-R) (first order), and 

• Ar(AsQ) = As (ArQ) = f (second order, a "delta of a 
delta query"). 

Then we can simultaneously maintain all these views using 
each other, using exclusively summation and avoiding the 
computation of any products. The fourth view is constant 
and does not depend on the database. Each of the other 
views is refreshed when a tuple is inserted by adding the 
appropriate delta view. For instance, as a tuple is inserted 
into R, we add ArQ to Q and An AsQ to AsQ. (No change 
is required to ArQ, since A_rA_rQ = 0.) Suppose R con- 
tains 2 tuples and S contains 3 tuples. If we add a tuple to 
S, we increment Q by 2 (AsQ) to obtain 8 and ArQ by 1 
(AsArQ) to get 4. If we subsequently insert a tuple into 
R, we increment Q by 4 (ArQ) to 12 and AsQ by 1 to 3. 
Two further inserts of S tuples yield the following sequence 
of states of the materialized views: 
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Again, the main benefit of using the auxiliary views is that 
we can avoid computing the product R x S (or in general, 
joins) by simply summing up views. In this example, the 
view values of the (fc + l)-th row can be computed by just 
three pairwise additions of values from the fc-th row. □ 



This is the simplest possible example query for which the 
viewlet transform includes a second-order delta query, omit- 
ting any complex query features (e.g., predicates, self-joins, 
nested queries). Viewlet transforms can handle general up- 
date workloads including deletions and updates, as well as 
queries with multi-row results. 

For a large fragment of SQL, higher-order IVM avotds 
join processing in any form, reducing all the view refresh- 
ment work to summation. Indeed, joins are only needed 
in the presence of inequality joins and nested aggregates in 
view definitions. The viewlet transform performs repeated 
(recursive) delta rewrites. Nested aggregates aside, each fc- 
th order delta is structurally simpler than the (fc — l)-th 
order delta query. The viewlet transform terminates, as for 
some n, the n-th order delta is guaranteed to be constant, 
only depending on the update but not on the database. In 
the above example, the second-order delta is constant, not 
using any database relation. 

Our higher-order IVM framework, DBToaster, realizes as- 
incremental-as-possible query evaluation over SQL with a 
query language extending the bag relational algebra, query 
compilation and a variety of novel materialization and opti- 
mization strategies. DBToaster bears the promise of provid- 
ing materialized views of complex long-running SQL queries, 
without window semantics or other restrictions, at very high 
view refresh rates. The data may change rapidly, and still 
part of it may be long-lived. A DDMS can use this func- 
tionality as the basis for richer query constructs than those 
supported by stream engines. DBToaster takes as input 
SQL view queries, and automatically incrementalizes them 
into procedural C++ trigger code where all work reduces to 
fine-grained, low-cost updates of materialized views. 



Example 2 Consider the query 

Q = select sum (LI .PRICE * O.XCH) 

from Orders 0, Lineltem LI where . ORDK = LI.ORDK; 

on a TPC-H like schema of Orders and Lineitem in which 
lineitems have prices and orders have currency exchange 
rates. The query asks for total sales across all orders weighted 
by exchange rates. We materialize the views for query Q as 
well as the first-order views Q_LI (A LI Q) and Q_0 (AoQ). 
The second-order deltas are constant w.r.t. the database and 
have been inlined in the following insert trigger programs 
that our approach produces for query Q. 

on insert into values (ordk, custk, xch) do { 
Q += xch * Q_0 [ordk] ; 
Q_LI[ordk] += xch; 

} 

on insert into LI values (ordk, partk, price) do { 
Q += price * Q_LI [ordk] ; 
Q_0[ordk] += price; 

> 

The query result is again scalar, but the auxiliary views are 
not, and our language generalizes them from SQL's multi- 
sets to maps that associate multiplicities with tuples. This 
is again a very simple example (more involved ones are pre- 
sented throughout the paper), but it illustrates something 
notable: while classical incremental view maintenance has to 
evaluate the first-order deltas, which takes linear time (e.g., 
(AoQ)[ordk] is select sum(LI .PRICE) from Lineitem LI 
where LI . 0RDK=ordk), we get around this by performing 
IVM of the deltas. This way our triggers can be evaluated 
in constant time for single-tuple inserts in this example. 

The delete triggers for Q are the same as the insert triggers 
with += replaced by -= everywhere. □ 

This example presents single-tuple update triggers. View- 
let transforms are not limited to this but support bulk up- 
dates. However, delta queries for single-tuple updates have 
considerable additional optimization potential, which our 
compiler leverages to create very efficient code that refreshes 
views whenever a new update arrives. We do not queue up- 
dates for bulk processing, and so maximize view availability 
and minimize view refresh latency, enabling ultra-low la- 
tency monitoring and algorithmic trading applications. 

On paper, higher-order IVM clearly dominates classical 
IVM. If classical IVM is a good idea, then doing it recur- 
sively is an even better idea. The same efficiency improve- 
ment argument in favor of IVM of the base query also holds 
for IVM of the delta query. Considering that joins are expen- 
sive and this approach eliminates them, higher-order IVM 
has the potential for excellent query performance. 

In practice, how well do our expectations of higher-order 
IVM translate into real performance gains? A priori, the 
costs associated with storing and managing additional auxil- 
iary materialized views for higher-order delta queries might 
be more considerable than expected. This paper presents 
the lessons learned in an effort to realize higher-order IVM, 
and to understand its strengths and drawbacks. Our contri- 
butions are as follows: 

1. We present the concept of higher-order IVM and describe 
the viewlet transform. This part of the paper generalizes 
and consolidates our earlier work [3, 19]. 

2. There are cases (inequality joins and certain nesting pat- 
terns) when a naive viewlet transform is too aggressive, 
and certain parts of queries are better re-evaluated than 
incrementally maintained. We develop heuristics and a 
cost-based optimization framework for trading off mate- 
rialization and lazy evaluation for best performance. 
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3. We have built the DBToaster system which implements 
higher-order IVM. It combines an optimizing compiler 
that creates efficient update triggers based on the tech- 
niques discussed above, and a runtime system (currently 
single-core and main-memory based 1 ) to keep views con- 
tinuously fresh as updates stream in at high data rates. 

4. We present the first set of extensive experimental re- 
sults on higher-order IVM obtained using DBToaster. 
Our experiments indicate that frequently, particularly 
for queries that consist of many joins or nested aggrega- 
tion subqueries, our compilation approach dominates the 
state of the art, often by multiple orders of magnitude. 
On a workload of automated trading and ETL queries, 
we show that current systems cannot sustain fresh views 
at the rates required in algorithmic trading and real-time 
analytics, while higher-order IVM takes a significant step 
to making these applications viable. 

Most of our benchmark queries contain features such as 
nested subqueries that no commercial IVM implementation 
supports, while our approach handles them all. 

2. RELATED WORK 

2.1 A Brief Survey of IVM Techniques 

Database view management is a well-studied area with 
over three decades of supporting literature. We focus on the 
aspects of view materialization most pertinent to DDMS. 

Incremental Maintenance Algorithms, and Formal 
Semantics. Maintaining query answers has been consid- 
ered under both the set [6, 7] and bag [8, 14] relational alge- 
bra. Generally, given a query on N relations Q(Ri, . . . , Rn), 
classical IVM uses a first-order delta query Ar 1 Q = Q(Ri U 
AJ?i, 7?2, • • • -Rjv) — Q(Ri, • • • , Rn) for each input relation 
Ri in turn. The creation of delta queries has been studied 
for query languages with aggregation [25] and bag seman- 
tics [14], but we know of no work that investigates delta 
queries of nested and correlated subqueries. [17] has con- 
sidered view maintenance in the nested relational algebra 
(NRA), however this has not been widely adopted in any 
commercial DBMS. Finally, [33] considered temporal views, 
and [22] outer joins and nulls, all for flat SPJAG queries 
without generalizing to subqueries, the full compositional- 
ity of SQL, or the range of standard aggregates. 

Materialization and Query Optimization Strategies. 

Selecting queries to materialize and reuse during processing 
has spanned fine-grained approaches from subqueries [27] 
and partial materialization [20, 28], to coarse-grained meth- 
ods as part of multiquery optimization and common subex- 
pressions [16, 35]. Picking views from a workload of queries 
typically uses the AND-OR graph representations from mul- 
tiquery optimization [16, 27], or adopts signature and sub- 
sumption methods for common subexpressions [35]. [27] 
picks additional views to materialize amongst subqueries of 
the view definition, but only performs first-order mainte- 
nance and does not consider the full framework (binding 
patterns, etc) required with higher-order deltas. Further- 
more, the optimal set of views is chosen based on the main- 
tenance costs alone, from a search space that can be doubly 
exponential in the number of query relations. 

Physical DB designers [2, 36] often use the query opti- 
mizer as a subcomponent to manage the search space of 
equivalent views, reusing its rewriting and pruning mecha- 
nisms. For partial materialization methods, ViewCache [28] 

1 This is not an intrinsic limitation of our method, in fact our 
trigger programs are particularly nicely parallelizable [19]. 



and DynaMat [20] use materialized view fragments, the for- 
mer materializing join results by storing pointers back to 
input tuples, the latter subject to a caching policy based on 
refresh time and cache space overhead constraints. 

Evaluation Strategies. To realize efficient maintenance 
with first-order delta queries, [9, 34] studied eager and lazy 
evaluation to balance query and update workloads, and as 
background asynchronous processes [29], to achieve a vari- 
ety of view freshness models and constraints [10] . Evaluat- 
ing maintenance queries has also been studied extensively 
in Datalog with semi-naive evaluation (which also uses first- 
order deltas) and DRed (delete-rederive) [15]. Finally, [13] 
argues for view maintenance in stream processing, which 
reinforces our position of IVM as a general-purpose change 
propagation mechanism for collections, on top of which win- 
dow and pattern constructs can be defined. 

2.2 Update Processing Mechanisms 

Triggers and Active Databases. Triggers, active database 
and event-condition-action (ECA) mechanisms [4] provide 
general purpose reactive behavior in a DBMS. The litera- 
ture considers recursive and cascading trigger firings, and 
restrictions to ensure restricted propagation. Trigger-based 
approaches require developers to manually convert queries 
to delta form, a painful and error-prone process especially 
in the higher-order setting. Without manual incremental- 
ization, triggers suffer from poor performance and cannot 
be optimized by a DBMS when written in C or Java. 

Data Stream Processing. Data stream processing [1, 24] 
and streaming algorithms combine two aspects of handling 
updates: i) shared, incremental processing (e.g. sliding win- 
dows, paired vs paned windows), ii) sublinear algorithms 
(i.e. polylogarithmic space bounds). The latter are approx- 
imate processing techniques that are difficult to program 
and compose, and have had limited adoption in commercial 
DBMS. Advanced processing techniques in the streaming 
community also focus almost entirely on approximate tech- 
niques when processing cannot keep up with stream rates 
(e.g. load shedding, prioritization [30]), on shared process- 
ing (e.g. on-the-fly aggregation [21]), or specialized algo- 
rithms and data structures [11]. Our approach to stream- 
ing is about generalizing incremental processing to (non- 
windowed) SQL semantics (including nested subqueries and 
aggregates). Of course, windows can be expressed in this 
semantics if desired. Similar principles are discussed in [13]. 
Automatic Differentiation and Incrementalization, 
and Applications. Beyond the database literature, the 
programming language literature has studied automatic in- 
crementalization [23], and automatic differentiation. Au- 
tomatic incrementalization is by no means a solved chal- 
lenge, especially when considering general recursion and un- 
bounded iteration. Automatic differentiation considers deltas 
of functions applied over scalars rather than sets or collec- 
tions, and lately in higher-order fashion [26] . Bridging these 
two areas of research would be fruitful for DDMS to support 
UDFs and general computation on scalars and collections. 

3. QUERIES AND DELTAS 

In this section, we fix a formalism for queries that makes 
it easy to talk cleanly and concisely about delta processing, 
and we describe the construction of delta queries. 

3.1 Data Model and Query Language 

Our data model generalizes multiset relations (as in SQL) 
to tuple multiplicities that are rational numbers. This for 
one allows us to treat databases and updates uniformly (for 
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Figure 1: Examples of union, join, and aggregation 
of generalized multiset relations with rational num- 
ber tuple multiplicities. 

instance, a delete is a relation with negative multiplicities, 
and applying it to a database means unioning/adding it to 
the database). It also allows us to use multiplicities to rep- 
resent aggregate query results (which do not need to be inte- 
gers). As a consequence, when performing delta-processing 
on aggregate queries, growing an aggregate means adding 
to the aggregate value rather than to delete the tuple with 
the old aggregate value and insert a tuple with the new ag- 
gregate value. Maintaining aggregates in the multiplicities 
allows for simpler and cleaner bookkeeping. It is a cosmetic 
change that does not keep us from supporting SQL queries. 

Formally, a thus generalized multiset relation (GMR) is a 
function from tuples to rational numbers, with finite support 
(i.e., only a finite number of tuples have nonzero multiplic- 
ity). The union, join, selection, and grouping-sum-aggregate 
operations are defined in the way that naturally generalizes 
the same operations on multiset relations: 
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RixiS 
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Here, n is the projection of records, rather than relations, 
removing fields of t whose labels are not among the column 
names A, and sch(_R) denotes the list of column names of 
GMR R. An aggregation Sum^T? almost works like the 

SQL query select A, sum(/) from R group by A, with 
the difference that SQL puts the aggregate value into a new 
column, while Sum^.^7? puts it into the multiplicity of the 
group-by tuple. Aggregations can also serve as multiplicity- 
preserving projections, and a query Sum^-. 1 (i?) corresponds 

equivalently to SQL queries select A from R and select 
A, sum(l) from R group by A. Examples of join, union, 
and aggregation are shown in Figure 1. 

Our query language includes relational atoms R, constant 
singleton relations, natural join, union (denoted +), selec- 
tion, grouping sum-aggregates, and column renaming p: 

Q ::= R I {A : a H- c} | Q 00 Q | Q + Q \ o^Q \ Sxan A . f Q \ p A Q 

where the c are rational numbers, / are terms, and <f> are 
conditions over terms. Terms are define using arithmetics 
over rational constants and column names. Additionally 
we can use non-grouping aggregates as terms (the value is 



the multiplicity), specifically in selection conditions. That 
way we can express queries with nested aggregates. Nested 
aggregates may be correlated with the outside in the way we 
are used to from SQL. For example, we write trc<Sum j4 . B i{S 
for the SQL query 

select * from S 

where S.C < (select sum(B) from R where R.A=S.A) 

We can perform deletions writing R — S, and this is no 
fundamentally new operation since we can define R— S := 
R + (S 1x1 {() i-> —1}). Here {() M> —1} is a miliary case of 
singleton GMR construction {A : a M> c}. 

There is no explicit syntax for universal quantification/re- 
lational difference or aggregates other than Sum, but all 
these features can be expressed using (nested) sum-aggregate 
queries (a popular homework exercise in database courses). 
Special handling of these features in delta processing and 
query optimization could yield performance better than what 
we report in our experiments. However, granting these de- 
finable features specialized treatment is beyond the scope of 
this paper. As a consequence, our implementation provides 
native support for only the fragment presented above and 
the experiment use only techniques described in the paper. 

We will use relational algebra and SQL syntax interchange- 
ably as we are used to from bag relational algebra and SQL. 

3.2 Computing the Delta of a Query 

We next show how to construct delta queries. The reader 
familiar with incremental view maintenance may skip this 
section, but note that the algebra just fixed has the nice 
property of being closed under taking deltas. For each query 
expression Q, there is an expression AQ of the same algebra 
that expresses how the result of Q changes as the database 
D is changed by update workload AD, 

AQ(D, AD) := Q(D + AD) - Q(D). 

Thanks to the strong compositionality of the language, 
we only have to give delta rules for the individual operators. 
These rules are given and studied in detail in [19]. In short, 



A(Qi +Q 2 ) 
A(Sum^ ;/ Q) 
A(Qi mQ 2 ) 

A{o 6 Q) 



(AQ,) + (AQ 2 ), 
Sum^ (AQ), 

((AQi) xi Q 2 ) + (Qi x (AQ 2 )) 
+ ((AQi)x(AQ 2 )), 
as (AQ). 



AR is the update to R. In the case that the update does 
not change 7? (but other relation(s)), AR is empty. 

Here we assume that / and 9 do not contain nested ag- 
gregates. Achieving the full generality is not a technical 
problem, but requires notation beyond the scope of this 
paper; we refer to [19] for the general case. We will see 
in Section 5 that, in practice, we will not need deltas for 
conditions with nested aggregates, as we will decide to re- 
evaluate rather than materialize and incrementally maintain 
such conditions. 



Example 3 Given schema R(AB), S(CD), and query 

select sum(A * D) from R, S where B = C 

or, in the algebra, SumQ-A*D(o'B=c(R x S)). Modulo names, 
this is the query of Example 2. The delta for this query as 
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we apply a change AR to relation R but leave S unchanged 
(AS is empty) is 

ASum 0; ^, D (a B= c(fiNS)) = 

Sum . A , D A(a B =c(R ex S)) = 

Sum {) . A , D (a B =cA(R tx S)) 

and by the delta rule for ex, 

A(R 1x3 5) = (Ai?) x S + i? ixi (AS) + (Ai?) tx (AS 1 ). 
Thus the delta query is SumQ : A*D(o'B=c((AR) tx S)). □ 

The delta rules work for bulk updates. The special case 
of single-tuple updates is interesting since it allows us to 
simplify delta queries further and to generate particularly 
efficient view refresh code. 

Example 4 We continue the Example 3, but now assume 
that Ai? is an insertion of a single tuple (A : x, B : y). The 
delta query Sum q ; a*d(cb=c ({(A : x, B : y}} IX S)) can be 
simplified to Sum^. „ D (a y= cS). □ 

3.3 Binding Patterns 

Query expressions have binding patterns: There are in- 
put variables or parameters without which we cannot eval- 
uate these expressions, and there are output variables, the 
columns of the schema of the query result. Each expres- 
sion Q has input variables or parameters x~t n and a set of 
output variables Xout, which form the schema of the query 
result. We denote such an expression as Q[a;Tn][a;oiit]- The 
input variables are those that are not range-restricted in a 
calculus formulation, or cquivalently have to be understood 
as parameters in an SQL query because their values cannot 
be computed from the database: They have to be provided 
so that the query can be evaluated. 

The most interesting case of input variables occurs in a 
correlated nested subquery, viewed in isolation. In such a 
subquery, the correlation variable from the outside is such 
an input variable. The subquery can only be computed if a 
value for the input variable is given. 

Example 5 We illustrate this by an example. Assume re- 
lation R has columns A, B and relation S has columns C, D. 
The SQL query 

select * from R 

where B < (select sum(D) from S where A > C) 

is equivalent to Sum*;i((T S< sum n . D (o- A>c (S))-R) m the al- 
gebra. Here, all columns of the schema of R are output 
variables. In the subexpression Sum() ;£) ((7A>c(S)), A is an 
input variable; there are no output variables since the ag- 
gregate is non-grouping. □ 

Also note that taking a delta adds input variables param- 
eterizing the query with the update. In Example 4 for in- 
stance, the delta query has input variables x and y to pass 
in the update. Delta queries for bulk updates have relation- 
valued parameters. 

4. THE VIEWLET TRANSFORM 

We are now ready for the viewlet transform. If we restrict 
the query language to exclude aggregates nested into con- 
ditions 2 (for which the delta query was complicated), the 
query language fragment has the following nice property. 

2 We will eliminate this restriction on the technique in the 
next section. 



AQ is structurally strictly simpler than Q when query com- 
plexity is measured as follows. For union(+)-free queries, 
the degree deg(Q) of query Q is the number of relations 
joined together. We can use distributivity to push unions 
above joins and so give a degree to queries with unions: the 
maximum degree of the union- free subqueries. Queries are 
strongly analogous to polynomials, and the degree of queries 
is defined precisely as it is defined for polynomials (where 
the relation atoms of the query correspond to the variables 
of the polynomial) . 

Theorem 1 ([19]) //deg(Q) > 0, then 
deg(AQ) = deg(Q) - 1. 

The viewlet transform makes use of the simple fact that 
a delta query is a query too. Thus it can be incrementally 
maintained, making use of a delta query to the delta query, 
which again can be materialized and incrementally main- 
tained, and so on, recursively. By the above theorem, this 
recursive query transformation terminates in the deg(Q)-th 
recursion level, when the rewritten query is a "constant" in- 
dependent of the database, and dependent only on updates. 

All queries, aggregate or not, map tuples to rational num- 
bers (= define GMRs). Thus it is natural to think of the 
views as map data structures (dictionaries). In this section, 
we make no notational difference between queries and (ma- 
terialized) views, but it will be clear from the context that 
we are using views when we increment them. 

Definition 1 The viewlet transform turns a query into a 
set of update triggers that together maintain the view of 
the query and a set of auxiliary views. Assume the query 
Q has input variables (parameters) x in . For each relation R 
used in the query, the viewlet transform creates a trigger 

on update R values D R do Tr. 

where Dr is the update - a generalized multiset relation - 
to the relation named R and Tr is the trigger body, a set of 
statements. The trigger bodies are most easily defined by 
their computation by a recursive imperative procedure VT 
defined below (initially, the statement lists Tr are empty): 

procedure VT(Q, x in ): 

foreach relation name R used in Q do { 

Tr += (foreach x in do Q[x in ] += A R Q[xi n D R ]) 
if deg(Q) > then { 

let D be a new variable of type relation of schema R; 
VT(A fl Q, x in D) 

} } 

Here, += on Tr appends a statement to an imperative code 
block, and += on generalized multiset relations uses the + of 
Section 3.1. Exactly those queries occurring in triggers that 
have degree greater than zero are materialized. Of course, 
these are exactly those queries that are added to by trigger 
statements: those that are incrementally maintained. □ 

Example 6 For example, if the schema contains two rela- 
tions R and S and query Q has degree 2, then VT(Q, ()) 
returns as Tr the code block 

Q += ArQ[Dr]; 

foreach D 1 do A H Q[Di] += ArArQ^, D R ]; 
foreach D 2 do A S Q[D 2 ] += ArA s Q[D 2 ,Dr] 

The body for the update trigger of S, T$, is analogous. Note 
that the order of the first two statements matters. For cor- 
rectness, we read the old versions of views in a trigger. □ 
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The viewlet transform bears a superficial analogy with 
the Haar wavelet transform, which also materializes a hi- 
erarchy of differences; however, these are not differences of 
differences but differences of recursively computed sums. 

At runtime, each trigger statement loops over a relevant 
subset of the possible valuations of the parameters of the 
views used in the statement. For relation- typed parame- 
ters, this is a priori astronomically expensive. There are 
various ways of bounding the domains to make this feasible. 
Furthermore, parameters can frequently be optimized away. 
Nevertheless, single-tuple updates offer particular optimiza- 
tion potential, and we focus on these in this paper. 

We will study single-tuple insertions, denoted +R(i) for 
the insertion of tuple t into relation R, and single-tuple dele- 
tions —R(t). Here we create insert and delete triggers in 
which the argument is the tuple, rather than a generalized 
multiset relation, and we avoid looping over relation-typed 
variables. 



Example 7 We return to the query Q of Example 4, with 
single-tuple updates. The query has degree two. The second- 
order deltas (A sgnHj j (Xiy )A sgnsS ( ZiU )Q)[a;, t/, z,u] have value 
sgn^sgng Sumn. im (iT t=z {()}), which is equivalent to (} i-> 
sgn^sgn^ if y = z then x * u else 0; here sgn^, sgn s G {+, — }. 
Variables x and y are arguments of the trigger and are bound 
at runtime, but variables z and u need to be looped over. 
On the other hand, the right-hand side of the trigger is only 
non-zero in case that y = z. So we can substitute z by y 
everywhere and eliminate z. Using this simplification, the 
on-insert into R trigger +R(x, y) according to the viewlet 
transform is the program 

Q += A +R {x, v )Q[x,y\; 
foreach u do A + s(y, u )C 



foreach u do A_ 



S(y,u) 



u)Q 



y,u 



+= {(} <-¥ x * u}; 
-= {(} <-¥ x * u} 



The remaining triggers are constructed analogously. The 
trigger contains an update rule for the (in this case, scalar) 
view Q for the overall query result, which uses the auxiliary 
view &.±R( x ,y)Q which is maintained in the update triggers 
for S, plus update rules for the auxiliary views A±s(z .u)Q 
that are used to update Q in the insertion and deletion trig- 
gers on updates to S. 

The reason why we did not show deltas A±r(,..)A±r(... )Q 
or A±s(... ) A±s(... )Q is that these are guaranteed to be 0, 
as the query does not have a self-join. 

A further optimization, exploiting distributivity and pre- 
sented in the next section, eliminates the loops on u and 
leads us to the triggers of Example 2. □ 

We observe that the structure of the work that needs to be 
done is extremely regular and (conceptually) simple. More- 
over, there are no classical large-granularity operators left, 
so it does not make sense to give this workload to a classical 
query optimizer. There are for- loops over many variables, 
which have the potential to be very expensive. But the 
work is also perfectly data-parallel, and there are no data 
dependencies comparable to those present in joins. All this 
provides justification for making heavy use of compilation. 

Note that there are many optimizations not exploited in 
the presentation of viewlet transforms. Thus, we will refer 
to the viewlet transform as presented in this section as naive 
recursive IVM in the experiments section. We will present 
improvements and optimizations next. 

5. OPTIMIZING VIEWLETS 

In this section, we present optimizations of the viewlet 
transform as well as heuristics and a cost model that allow 
us to avoid materializing views with high maintenance cost. 



Query Decomposition 

M(Svm A g. fittf2 (Qi x Qi)) => 

>1(Sum A/i (Q 1 ))M^l(Sum 5:/2 (Q 2 )) (1) 
Qi and Q 2 have no common columns 
A and B are the group-by terms of each 

Factorization and Polynomial Expansion 

M(Q L tx (Qi + Q 2 + ■ ■ ■) tx Qr) & 

M(Q L txiQ 1 txiQ R ) + M(Q L txQ2txQR) + ..- (2) 

Input Variables 

M(Sum A . nSd) (a 0(Se) (Q))) => 

Sum 4;/(BC) ( a e(BC) ( M ( Sum AS;l 02)))) (3) 

/, 9 arc functions over terms 

A is the group-by variables of the aggregate over Q 

B is the output variables of Q used by /, 9 

C is the input variables that do not appear in Q 

Nested Aggregates and Decorrelation 

M{Sum x . f {a e(QNS) (Qo))) => 

Sum A:l( a 0(M(Q N ),B)( M ( Sum AB:f(Qo)))) (4) 

Qjv is a non-grouping aggregate term 
/, 9 are functions over terms 

A is the group-by variables of the aggregate over Qo 
B is the output variables of Qo used by /, 9 and Qjv 

Figure 2: Rewrite rules for partial materialization. 
Bidirectional arrows indicate rules that are applied 
heuristically or using the cost model from Section 5.1. 

5.1 Materialization Decisions 

For any query Q, the naive viewlet transform produces a 
single materialized view Mq. However, it is frequently more 
efficient to materialize Q piecewise, as a collection of incre- 
mentally maintained materialized views Mq and an equiva- 
lent query Q' that is evaluated over these materialized views. 
We refer to the rewritten query and its piecewise maps as a 
materialization decision for Q, denoted (Q',Mq). 

DBToaster selects a materialization decision for a query 
Q iteratively, by starting with the naive materialization de- 
cision ((Mq z i),(Mq,i := Q)) and applying several rewrite 
rules up to a fixed point. These rewrite rules are presented 
in Figure 2, and are discussed individually below. Figure 
3 shows the applicability of these rules to the experimental 
workload discussed in Section 6 and Appendix A. 

For clarity, we will use a materialization operator M to 
show materialization decisions juxtaposed with their corre- 
sponding queries. For example, one possible materialization 
decision for the query Q :— Qi IX Q2 is: 



M (Qi) x M(Q 2 ) = ((M Q)1 cxi M Q , 2 ), {Mq, 



?>}>■ 



We will first discuss the use of these rules in a heuristic 
optimizer, which when applied as aggressively as possible 
produces near-optimal trigger programs for maintaining the 
vast majority of queries. We also briefly discuss a cost-based 
optimization strategy to further improve performance. 
Duplicate View Elimination. As the simplest optimiza- 
tion, we observe that the naive viewlet transform produces 
many duplicate views. This is primarily because the delta 
operation typically commutes with itself; ArAsQ = As ArQ 
for any Q that does not contain nested aggregates over R or 
S. Even simple structural equivalence effectively identifies 
this sort of view duplication. View de-duplication substan- 
tially reduces the number of views created. 
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Query Decomposition. DBToaster makes extensive use 
of the generalized distributive law [5] (which plays an impor- 
tant role for probabilistic inference with graphical models) 
to decompose the materialization of expressions with dis- 
connected join graphs. This rule is presented in Figure 2.1. 

If the join graph of Q includes multiple disconnected com- 
ponents Qi, Q 2 , ■ ■ ■ (i.e., Q :— Qi x Q 2 x . . .), it is better 
to materialize each component independently. The cost of 
selecting from, or iterating over Q is identical for both ma- 
terialization strategies. Furthermore, maintaining each indi- 
vidual Qi is less computationally expensive; the decomposed 
materialization stores (and maintains) only \ Qi\ values, 
while the combined materialization handles \\ i \ Qi\ values. 

This optimization plays a major role in the efficiency of 
DBToaster, and in the justification of compilation. Taking 
a delta of a query with respect to a single-tuple update re- 
places a relation in the query by a singleton constant tuple, 
effectively eliminating one hyperedge from the join graph 
and creating new disconnected components that can be fur- 
ther decomposed. Query decomposition is also critical for 
ensuring that the number of maps created for any acyclic 
query is polynomial. 

Polynomial Expansion and Factorization. As described 
above, query decomposition operates exclusively over con- 
junctive queries. In order to decompose across unions, we 
observe that the union operation can be pushed through ag- 
gregate sums (i.e, Sum(Qi + Q2) = Sum(Qi) + Sum(<22)), 
and distributed over joins. 

Any query expression can be expanded into a flattened 
polynomial representation, which consists of a union of purely 
conjunctive queries. Query decomposition is then applied to 
each conjunctive query individually. This rewriting rule is 
presented in Figure 2.2. 

Note that this rewrite rule can also be applied in reverse. 
A polynomial expression can be factorized into a smaller rep- 
resentation by identifying common subexpressions (Ql and 
Qr in the rewrite rule) and pulling them out of the union. 
The cost-based optimizer makes extensive use of factoriza- 
tion to fully explore the space of possible materialization 
decisions. The heuristic optimizer does not attempt to fac- 
torize while making a materialization decision. However, 
after a materialization decision {Mq, {. . .}) is finalized, the 
materialized query Mq itself is simplified by factorization. 

Input Variables. Queries with input variables have in- 
finite domains and cannot be fully materialized. By de- 
fault, DBToaster's heuristics ensures that input variables 
are avoided, in the query and all its subexpressions. 

Input variables are originally introduced into an expres- 
sion by nested aggregates, and as part of deltas. They ap- 
pear exclusively in selection and aggregation operators. 

The rewrite rule shown in Figure 2.3 ensures that mate- 
rialized expressions do not have input variables by pulling 
operators with input variables out of the materialized ex- 
pression. If an operator can be partitioned into components 
with output variables only and components with input vari- 
ables, only the latter are pulled out of the expression. 

In addition to this heuristic approach, the cost-based op- 
timizer explores alternative materialization decisions where 
some (or all) of the input variables in an expression are kept 
in the materialized expression. With respect to Figure 2.3, 
these input variables are treated as elements of A instead of 
C. At runtime, only a finite portion of the domain of these 
input variables is maintained in the materialized view. 

A materialized view with input variables acts, in effect, as 
a cache for the query results. Unlike a traditional cache how- 
ever, the contents of this map are not invalidated when the 
underlying data changes, but instead maintained incremen- 



tally. These sorts of view caches are analogous to partially 
materialized views [20, 28]. 

View caches are only beneficial when the size of the active 
domain of an input variable is small, and so the heuristic 
optimizer does not attempt to create them 3 . 

Deltas of Nested Aggregates. Thus far, we have ignored 
queries containing nested subqueries. When the delta of the 
nested subquery is nonzero, the delta of the entire query is 
not simpler than the original. The full delta rule for nested 
presented in [19] effectively computes the nested aggregate 
twice: once to obtain the original value, and once to obtain 
the value after incorporating the delta. 

Example 8 Consider the following query, with relations R 
and S with columns A and B respectively: 

Q ■ = Sum<) ; i((TSum <> . 1 (S)=A(-R)) 

By the delta rule for nested aggregates, 

A+s(B') := Sum ( ) ;1 (cr Sum(>!l(S ) + i )=A (i?)) - 
Sum <); i((j Sum0;l{s)=A (i?)) 

Because the original nested query appears in the delta 
expression, the naive viewlet transform will not terminate 
here. To address this, the delta query is is decorrelated into 
separate materialized expressions for the nested subquery 
and the outer query. The rewrite rule of Figure 2.4 is applied 
twice (once to each instance). Each materialized expression 
now has a lower degree than the original query. 

Although this rule is necessary for termination, it intro- 
duces a computation cost when evaluating the delta query. 
Note however, that this rule is only required (and thus, only 
used) when the delta of the nested subquery is nonzero. 

Example 9 Continuing Example 8, the materialization de- 
cision for A + 5(g/)Q uses two materialized views: 
A^Q.i Sum/\. 1 (S') and Mq^ '■= R, and uses each twice. 
However, A +R (a')Q naturally has a lower degree than Q, 
and is thus materialized in its entirety. 

Additionally, we observe that for some queries, it is more 
efficient to entirely recompute the query Q on certain up- 
dates. Consider the general form of a nested aggregate: 

Q ■= Sum y? . /i (cr(Sum y?5;/2 (Qo))) 

The delta of Q evaluates two nearly identical expressions. 
Naively, computing the delta costs twice as much as the orig- 
inal query and so re-evaluation is more efficient. However, if 
the delta's trigger arguments bind one or more variables in 
B, then the delta query only aggregates over a subset of the 
tuples in Qo and will therefore be faster. Based on this anal- 
ysis, the heuristic optimizer decides whether to re-evaluate 
or incrementally maintain any given delta query. 

Cost Model. While DBToaster's full cost-based optimizer 
is beyond the scope of this paper, we now briefly discuss 
its cost model. The dominant processing overheads are: (1) 
Updating maps with query results, and (2) Sum aggregates, 
which compound tuples into aggregate values. 

The cost of a materialization decision {Q' , Mq) includes 
both an evaluation component (cost e ) and a maintenance 
component (cost m ) for both the view materialized for the 
original query and all of the higher-order views. 

3 View caching is required for any materialized expression 
without finite support. This includes some forms of nested 
aggregates without input variables. A precise characteriza- 
tion of these expressions is beyond the scope of this paper. 
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Figure 3: Rewrite rules applied to our workload. 



DBToaster uses standard cardinality estimation [12, 32] 
for the number of distinct tuples when projecting the result 
of query Q down to columns A. We refer to this as the size 
of the domain of A in Q (|dom^(Q)|). If A is the full set of 
output variables of Q, then we refer to this as the complete 
domain of Q (|dom(Q)|). 

Let Q be the set of all subexpressions of Q. The cost of 
query Q is the sum of the sizes of the complete domains of 
the outer query Q and all queries nested immediately inside 
aggregate sums Qi in Q: 

cost e (Q) = |dom(Q)| + |dom(Qi)| 

{Qi|Sum(Q i )eQ} 

The maintenance cost of Q is based on the cost of main- 
taining all the MQ t i £ Mq. Every Mq,; must be updated for 
every change to a relation Rj that appears in it. If the delta 
query of Mq^ is materialized using materialization decision 

(Q' i j,MQ i .), and the rate of insertions into Rj is rate^., 
then the cost of maintaining Mq^ is 

cost m (Mg,j) — y^rate_ R .-cost e (Q' J )+ ^ cost m (M) 

R i Mel ^Qi,j 

This definition recurs on the cost of maintaining the maps 
required to evaluate Qi,j. The maintenance cost of a map 
that is already being materialized by another query is zero. 

The full cost of processing query Q is now 

cost(Q) = (rate rc f ros h ■ cost e (Q')) + cost m (MQ,i)) 

i 

where the refresh rate of Q depends on how frequently a 
fresh view must be made available. For the typical usage 
scenario a refresh occurs on every update, and rate re frcsh = 
J2j r&te Rj . 

5.2 Optimized Viewlet Transform Example 

Figure 4 shows the trigger program compiled from query 
Q18 from our test workload (see Appendix A). 

For simplicity, we use the condensed schema C(CK), 
0{CK,OK), and LI (OK, QTY). The query Q\\[CK] is: 

SuniCK:QTY ( 

a WO<Sum . QTY ,(a OK = OK ,(p OK , QTY ,LI))(C XI O 1X1 LI)) 

Due to space limitations we only show the derivation of in- 
sertions into Orders O and Lineitem LI. Insertions into 
Customer C are a simple extension, while deletions are du- 
als of insertions and are omitted entirely. 

Insertions into Orders. The first-order delta of Q for 
insertion of a single tuple (CK : ck, OK : ok) is 

(C CXI LI)) 

where Q ns =(100 < Sum 0; Q Ty / (p ok=OK ,p OK > tQT Y' LI)) 



on insert into C values (ck) do { 

01 Q[][ck] += Q c [][ck] 

02 foreach OK do Q LI [][ck,OK] += Q L l,c[][ck, OK] 

03 Qoi[][ck]+=l 

} 

on insert into values (ck,ok) do { 

04 Q[][ck] += Qoi[][ck]txQ O 2[][ok]<xicr 100<Qo2[][ok] (l) 

05 Q L i[][ck,ok] += Qoi[][ck] 

06 Q L i,c\\[ck,ok] += 1 

07 Qc\\[ck] += Q 2[][ok] IX <x 10 o<Q 02 [][ fc](l) 

} 

on insert into LI values (ok.qty) do { 

08 foreach CK do 

Q[][CK] += Q LI [][CK,ok] tx ( 
((Qo2[][ok] + {<> i y qty}) M o- 1Q0<qty+Qo2[][ok] (l)) 
- (Qo2[][ok] M CT 100<Qo2 [ ][ofe] (l))) 

09 foreach CK do 

Qc[][CK] += Q LI:C \}[CK,ok]tx( 

{{Qo2\}{ok] + {(> i ^ qty}) ixi cr 100<q ty + Q O2 [][ok]W) 

- (Qo2[][ok] cx<r 100<Qo2[][ofc] (l))) 

10 Qo2[][ok] += qty 

} 

Figure 4: DBToaster insert trigger program for Q18. 

By rewrite rule 1, this delta expression can be decomposed 
into two separate maps, since C and LI share no common 
columns. Furthermore, the nested subexpression does not 
contain relation O, so we do not apply rewrite rule 4 here. 
The delta expression can be materialized as follows: 

M(Sum(CK):i(o- C K=ckC)) tx] M(SumQ.Q TY ( 

0~OK = okA(100<Sum ( y QTY ,(tT ok = OK , (Pqk'.QTY' 

LI)))LI)) 

The second materialized map can be simplified further by 
rules 1 and 4. OK is bound to trigger parameter ok, which 
breaks the join graph between the selection predicate and 
LI. Then, since the selection predicate is being applied to 
a singleton, we can safely materialize only the aggregate in 
the predicate. Applying these optimizations gives us the 
following materialization decision (with 1 := {(} t-¥ 1}): 

M(Sum (CK) . 1 (a C K=ckC)) CX 

M(Sum {) . QTY (o-oK=okLI)) tx 
Sum (> . 1 (o- 1(K , <A1( s um(>;QTy((Tofc=OK L/))(l)) 

Trigger statement 04 uses the following set of views (note 
that Q02 is used twice): 

Qoi ■= Sum CK; i(C) Q02 ■= Sum K;QTY(LI) 

Qoi\\[CK] is maintained on insertions into C with: 

A +c(ck) Qoi :={(CK :ck)^l} 

which corresponds to trigger statement 03. Qo2[][Oif] is 
maintained similarly with trigger statement 10. 

Insertions into Lineitem. The first-order delta of Q for 
insertion of a single tuple (OK : ok, QTY : qty) is 

A+LI(ok,qty)Q '■= SumcK;QTY(o~OK=okA100<qty+Q nB ( 

C ex O tx (LI + {(OK : ok, QTY : qty) i-S- 1})) 

— ctok =ofcA100<Q„,, (C XI O CX LI)) 

When computing the above delta, we extend the delta rule 
of [19] for nested aggregates. The delta of a (decorrelated) 
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Figure 5: DBToaster performance overview. Note the 
logscale on the y-axis. (*) For VWAP, where DBToaster 
uses naive recursive compilation, we compare against a 
strategy that avoids input variables. 

nested aggregate loops over the full domain of that aggre- 
gate even when only a small subset of the domain is af- 
fected. We can exploit this to range-restrict variables of 
the nested subquery. In this example, this is done by the 
predicate OK = ok. If the nested subquery is correlated 
with the outer query on an equality, this range-restriction 
is propagated to the outer level, significantly reducing the 
computational cost. 

Repeated application of rewriting rules 2, 3, 4 and 1 with 
the trigger variable optimization, and pushing down selec- 
tions results in the following materialization decision: 

A4(SumcK;i(o-ox=ofc(C IX O))) tx ( 

(M(Q 2 ) + {<) H> qty}) cxi a W o< q ty+M(Q 2 )( 1 ) 
-M(Q 2 ) tx <t W0<M (q 2 )(1)) 

where Q2 = S\xmQ ; Q TY ((TOK=okLI) 

Apart from the outermost materialization (of C IX O), 
the remaining four materializations in this expression are not 
only equivalent, but identical to Q02, which is already being 
maintained. Only one view: Qli ■= 'S>wacK;i{(JOK-ok{C IX 
O)) is materialized. Rewriting the materialization decision 
produces trigger statement 08. 

Note that this statement requires a loop. We update Q 
iterating over domcif (A+liQ) = domcK(QLi)- For this 
example, the loop never encounters more than one tuple 
due to the foreign key dependency from O to C. 

Qli can be maintained in a manner analogous to that of 
Example 3, resulting in trigger statements 03, 05, and 06. 

6. EXPERIMENTAL RESULTS 

The DBToaster compiler produces trigger programs as 
C++ code with views implemented by Boost Multilndexes, a 
flexible main-memory collection data structure supporting 
a variety of secondary index types. Our compiler internals 
are ongoing research and outside the scope of this paper. 
We evaluate the experimental performance of DBToaster on 
Redhat Enterprise Linux with 16 GB of RAM, and an Intel 
Xeon E5620 2.4 GHz processor (on a single-core). 
Data and Query Workload. Our workload captures al- 
gorithmic order book trading and online business decision 
support scenarios that involve computing a variety of statis- 
tics to guide actions. Figure 6 lists the processing properties 
of our workload, with SQL code in Appendix A. 

The financial queries VWAP, MST, AXF, BSP, PSP, and 
BSV were run on a 2.63 million tuple trace of an order book 
update stream, representing one day of stock market activ- 
ity for MSFT. These are updates to a Bids and Asks table 
with a schema of a timestamp, an order id, a broker id, a 
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Figure 6: Features of the algorithmic trading and online 
decision support workloads used for experiments. 
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Figure 7: Comparison between DBToaster and two 
commercial query engines (in refreshes per second). 
Both the DBMS (DBX) and stream system (SPY) 
columns show the cost of full refresh on each update. 

price, and a volume. The TPC-H benchmark queries Q3, 
Qli, Q17, Q18, Q22, and SSB4 were run on a stream of up- 
dates adapted from a database generated by DBGEN[31]. 
We simulate a system that monitors a set of "active" orders 
by randomly interleaving insertions on all relations and in- 
jecting random deletions on Orders rows to keep the Orders 
table at around 30 thousand tuples. Results presented in 
Figures 8, 9 and 10 are based on a scaling factor 0.1 (100 
MB) database. We show that these results scale to longer 
streams in Section 6.2. 

To evaluate our compilation algorithm, DBToaster pro- 
duces three alternatives by terminating recursive compila- 
tion early. Depth compilation corresponds to re-evaluating 
the query on every update, while compilation at Depth 1 is 
classical first-order IVM. As the third option, DBToaster 
materializes as much of the query as possible (Naive Re- 
cursive), creating view caches and employing partial mate- 
rialization to decorrelate nested subqueries. Our results 
show the number of tuples processed by queries run over a 
replayed stream for a 1 hour period. 

6.1 Higher- Order IVM Performance 

We now analyze the steady-state performance of DBToaster. 
Comparison with Commercial Systems. Figure 7 com- 
pares higher-order IVM to the performance of a commer- 
cial DBMS (DBX) and stream processor (SPY) , anonymized 
due to their license agreements. We present a summary of 
our findings, an in-depth itemized breakdown of overheads 
is outside the scope of this paper. SPY does not support 
IVM, so the number presented is for full re-evaluation of the 
query on every update. For nested queries, our workload uti- 
lizes SPY's in-memory tables which significantly contributes 
to the performance gap from DBToaster due to their syn- 
chronization requirements in an asynchronous stream en- 
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Figure 8: (a) Q3, (b) Qll, (c) Q17, (d) Q18; (a) A 3-way linear join, (b) A 2-way linear join, (c) A 2-way join 
with an equality-correlated nested aggregate, (d) A 3-way join with an equality-correlated nested aggregate. 
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Figure 9: (a) Q22, (b) SSB4, (c) VWAP, (d) BSV; (a) A single table with an equality- and an inequality- 
correlated nested aggregates. Insertions into the Customer relation complete within the first 10% of the 
stream, (b) A 3-way star join with a maximum join width of 6. (c) A single table with an inequality- 
correlated and an uncorrelated nested aggregate. DBToaster chooses the naive recursive approach, so we 
compare against an approach that aggressively avoids input variables, (d) A 2-way self-join. 
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Figure 10: (a) PSP, (b) MST, (c) AXF, (d) BSP; (a) A 2-way join with two uncorrelated nested aggregates, 
(b) A 2-way join with two uncorrelated, and two inequality-correlated nested aggregates. None of the tested 
engines completed the trace within the 60 minute limit, (c) A 2-way inequality join, (d) An inequality 
self-join. 
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gine. For DBX, although this does support IVM, more than 
half of the queries in our test workload require features of 
SQL that cannot be handled incrementally by DBX's views 
subsystem. In our experiments, we found two significant 
contributors to DBX's overheads. First, because DBX only 
performs IVM after commits, transaction overheads greatly 
add to the cost of achieving fast refresh. Second, maintain- 
ing catalog information across many tables for high-rate up- 
dates also substantially impacts latencies and throughput. 
Equijoins. Q3 and Qll (Figure 8a, b) are 2- and 3-way lin- 
ear joins respectively, SSB4 (Figure 9b) is a 3-way star join 
with a maximum join-width of 6, and BSV (Figure 9d) is 
a 2-way self-join. As there are no inequalities, DBToaster 
and Naive Recursive Compilation produce mostly identical 
results. In the case of many 2-way joins, the first level deltas 
are very nearly the base relations, and so on Qll, IVM is 
able to perform as effectively as DBToaster. On BSV how- 
ever, DBToaster gets a substantial performance improve- 
ment by representing the materialized delta view with only 
a single aggregate value, making the update cost constant. 
SSB4 normally has a join width of 6. However, because the 
contents of the Nation table are static, DBToaster does not 
attempt to materialize any deltas needed to support updates 
to Nation, reducing the join width to 4 and eliminating sev- 
eral maps with high maintenance costs. 
Nested Aggregates. Q17 and Q18 (Figure 8c, d) are multi- 
way join nested-aggregate queries with simple nested aggre- 
gates, in both case with the nested aggregate correlated on 
an equality. Here, DBToaster's strong performance comes 
from decorrelating the nested subquery for only the deltas 
of Lineitem (on which both nested subqueries are based) . 

Q22 (Figure 9a) includes two nested aggregates, an uncor- 
rected aggregate on Customer that is compared against on 
the top level using an inequality and an equality-correlated 
aggregate on Orders that is compared against using an in- 
equality. The first nested subquery causes DBToaster to 
choose a strategy of re-evaluating the top-level query since 
the delta of the subquery with respect to updates to Cus- 
tomer is not simpler than the original subquery. The second 
subquery by itself would not have made this necessary since 
we can decorrelate it (due to the absence of both inequalities 
and the Customer relation) and avoid input variables in any 
query subexpression. Nevertheless, the two subqueries as 
well as the top-level aggregation without the inequality can 
be materialized, reducing re-evaluation to a loop over na- 
tions. This is seen in the performance graph as the query's 
slow startup ends once the last customer has been inserted. 

VWAP (Figure 9c) has a nested aggregate correlated on 
an inequality. The small domain of the correlation variable 
(price) makes this an ideal candidate for view caching. 

PSP (Figure 10a) has two uncorrelated nested aggregates. 
This query benefits from full re-evaluation on each execu- 
tion. However, polynomial expansion actually enables a 
graph decomposition that splits the query into 4 parts: 2 
constant time components and 2 independent linear time 
components in the number of distinct values of the column 
being compared to the nested aggregate (volume). 

MST (Figure 10b) is fundamentally similar to PSP, but 
rather than comparing its uncorrelated aggregates against 
columns from the base relations, they are each compared 
against another nested aggregate correlated on an inequality. 
This is a worst case scenario for DBToaster, as it cannot 
incrementally process this query in better than 0(n 2 ) time 
without specialized indexes (e.g., aggregate range trees). 
Inequijoins. AXF and BSP are both 2-way joins (Figure 
10c, d), with BSP being a self-join. In the case of AXF, both 
the join variable (price) and one of the aggregate variables 
(volume) are treated as input variables by naive recursive 




Figure 11: Performance scaling on the TPCH queries. 

materialization. In BSP, the join variable (t) has an ex- 
tremely large domain. In both cases, partial materialization 
outperforms naive. Since both are 2-way joins, IVM is nearly 
optimal - DBToaster achieves a small speed boost in both 
cases by not materializing the entire base relation. 

6.2 Working State Scalability 

Figure 11 analyzes the performance scaling properties of 
DBToaster on larger datasets and subsequently a larger work- 
ing state for its main-memory data structures. An update 
stream was synthesized from databases created by DBGEN 
at scaling factors 0.5, 1, 5, and 10 (500MB, 1GB, 5GB, and 
10GB respectively). As before, the Orders relation is kept 
at 30 thousand tuples. The Customer, Part, Supplier, or 
Partsupp are inserted completely and never deleted. With 
the exception of Q22, performance stays roughly constant 
as stream length grows. 

The running time of Q22 is dominated by the first 10% of 
the stream in each case, before the customers table has been 
fully inserted. The cost of each insertion into the customer 
table is linear in the size of the customer table. After all 
customer tuples have been loaded in, performance returns 
to a constant 35000 tuples per second, regardless of scale. 

7. CONCLUSION 

We presented a compiler and optimizer framework for 
higher-order IVM that uses aggressive simplification of re- 
cursive delta queries and a plethora of materialization strate- 
gies to make recursive IVM viable. Our compilation method 
is effective on a wide range of select-project-join-aggregate 
queries, including those with nested subqueries that are not 
supported by current IVM mechanisms. Our methods pro- 
vide scalable view refresh rates, often orders of magnitude 
over today's tools, providing the basis for our vision of DDMS. 
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SELECT b.broker_id, sum(a.volume-b. volume) 

FROM Bids b, Asks a 

WHERE b.broker_id = a.broker_id 

AND (a.price-b.price > 1000 OR b.price-a. price > 1000) 
GROUP BY b.broker_id; 

SELECT x.broker_id, sum (x . volume*x . price - y.volume*y. price) 
FROM Bids x, Bids y 

WHERE x.broker_id=y.broker_id AND x.t>y.t 
GROUP BY x.broker_id; 

SELECT x.broker_id, sumCx . volume*x .price*y . volume*y .price*0 . 5) 
FROM Bids x, Bids y 

WHERE x.broker_id = y.broker_id GROUP BY x.broker.id; 
SELECT b.broker_id, sumCa.price*a. volume - b.price*b. volume) 
FROM Bids b, Asks a 

WHERE 0.25* (select sum(al . volume) from Asks al) > 

(select sum(a2 . volume) from Asks a2 where a2 . price>a. price) 

AND 0.25* (select sum(bl . volume) from Bids bl) > 

(select sum(b2 . volume) from Bids b2 where b2 . price>b .price) 

GROUP BY b.broker_id; 

SELECT sum(a. price - b. price) FROM Bids b, Asks a 
WHERE b.volume>0.0001*(select sum(bl . volume) from Bids bl) 
AND a. volume>0 . 0001* (select sum(al .volume) from Asks al) ; 
SELECT sum(bl. price * bl. volume) FROM Bids bl 
WHERE 0.25 * (select sum(b3. volume) from Bids b3) > 
(select sum(b2 . volume) from Bids b2 
where b2 .price>bl .price) ; 
.orderkey, o.orderdate, o . shippriority, 
sumCli . extendedprice * (1 - li . discount) ) 
Customer c, Orders o, Lineitem li 
c.mktsegment = 'BUILDING' 
. custkey 
o . orderkey 
DATEC 1995-03-15') 
DATEC1995-03-15') 
o.orderdate, o . shippriority ; 
SELECT ps.partkey, sum(ps . supplycost * ps.availqty) 
FROM Partsupp ps , Supplier s 

WHERE ps . suppkey = s . suppkey GROUP BY ps . partkey ; 
SELECT sum(l. extendedprice) FROM Lineitem 1, Part p 
WHERE p. partkey = 1. partkey 

AND 1. quantity < 0.005 * (select sum(12 . quantity) 

from Lineitem 12 where 12. partkey = p. partkey); 
SELECT c. custkey, sumdl .quantity) 
FROM Customer c, Orders o, Lineitem 11 
WHERE 1 <= (select sum(l) where 

100 < (select sum(12 . quantity) from Lineitem 12 
where 11. orderkey = 12 . orderkey) ) 
AND c. custkey = o. custkey AND o. orderkey = 11. orderkey 
GROUP BY c. custkey; 

SELECT cl .nationkey, sum(cl . acctbal) FROM Customer cl 
WHERE cl. acctbal < 

(select sum(c2 . acctbal) from Customer c2 where c2 . acctbal>0) 
AND 0=(select sum(l) from Orders o where o . custkey=cl . custkey) 
GROUP BY cl. nationkey; 

SELECT sn.regionkey, cn.regionkey, p. type, 
sumCli . quantity) 

Customer c, Orders o, Lineitem li, 
Part p, Supplier s, Nation cn, Nation sn 
c. custkey = o. custkey 
o. orderkey = li. orderkey 
p. partkey = li. partkey 
s. suppkey = li. suppkey 
o.orderdate >= DATE( ' 1997-01-01 ' ) 
o.orderdate < DATE( ' 1998-01-01 ' ) 
cn. nationkey = c. nationkey 
sn. nationkey = s. nationkey 



SELECT 

FROM 

WHERE 

AND o . custkey = 

AND li. orderkey = 

AND o.orderdate < 

AND li.shipdate > 

GROUP BY o. orderkey, 



FROM 

WHERE 
AND 
AND 
AND 
AND 
AND 
AND 
AND 



GROUP BY sn.regionkey, cn.regionkey, p. type; 
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